The concept of stitching multiple camera images together in order to compose a wide field of view image is known, as is the concept of capturing multiple video signals to compose a panoramic or omni-directional image, with some stereographic functionality. See, for example, U.S. Pat. Nos. 5,703,604, 6,323,858, 6,356,397, 6,392,699, and 7,015,954 and US Patent Application No. 2003/0117488.
There are three general techniques for capturing omni-directional and/or stereographic images. In one technique, a camera is rotated using a servo-mechanism to image a spherical area of interest. This technique suffers from three significant drawbacks. First, the speed of image capture is limited by the rotational speed of the servo-mechanism and inertia of the assembly. This can place significant performance limits on the frame-rate and shutter-rate of the system as well as the speed with which users can scan the surroundings. Second, reliance on moving elements for operation inherently possesses greater maintenance requirements and suspect reliability. Third, multiple users of such a system are constrained to view the same part of the scene simultaneously, since only one direction can be viewed at a time.
In another technique, a single camera captures a wide field-of-view image (up to a full hemisphere) using a specially shaped optical element (usually a convex lens or mirror). This technique is actually a relatively ubiquitous method of capturing panoramic images. However, while this approach may be affordable and relatively prevalent, it also suffers from a number of significant drawbacks.
Because the entire scene is being captured by a single CCD (or similar image capture element), the total average information per pixel is significant, causing resolution loss. More importantly, however, because this technique generally involves projecting a spherical surface onto a flat, rectangular image capture element (e.g. the CCD or CMOS chip), significant distortion is unavoidable, negatively impacting resolution. Some amount of distortion can be processed out, but the information lost due to this inefficient image capture mechanism cannot be retrieved.
Two types of obscuration also occur using this method of imaging. The first occurs in designs where the optical element is a convex mirror, which eliminates the ability to capture the cone directly above or below the imager. The second occurs in cases where this technique is used to capture stereoscopic panoramic images, because each of the cameras obscures the other, laterally. Additionally, to use this approach, scenes must be well lit to obtain premium image quality.
In a third technique, images from multiple static cameras that cover the omni-directional space are stitched together. This technique provides key advantages over the previous techniques. Using multiple CCDs (or other image capture elements) to capture the entire omni-directional area increases the overall resolution of the image. Also, with the widespread use of digital imagers (digital cameras, camera phones, etc.) the cost of CCD & CMOS imaging components is rapidly decreasing, increasing the affordability of this approach. Additionally, use of more cameras, each with smaller field of view lenses minimizes distortions and the associated impact on resolution. Further, statically locating each camera improves the reliability and lowers the required maintenance of the design.
The main drawback in using this approach is the requirement it places on processing bandwidth. Simultaneously capturing and displaying high resolution, high frame rate (e.g. 30 FPS) images requires very high data bandwidths, approaching and possibly exceeding 1 GByte per second. If significant real-time video processing is also required, the bandwidth demands increase.
Prior art devices that create an image of the spherical surroundings by stitching together the images from multiple camera images have certain drawbacks. Most of these devices do not make use of more than 11 cameras and, thus, resolution suffers. In addition, to cover the same area with fewer cameras requires wide field of view lenses, causing distortion-induced resolution loss, as described above in connection with the use of fish-eye lenses for panoramic image capture. Obscuration is an issue in most of these prior designs when it comes to stereo capture. In particular, some of the camera systems are only able to grab stereo images by placing two of their omni-directional imagers adjacent to each other. Using such a set-up to capture panoramic wide field of view scenes is problematic as there is no easy way to remove the obscuration that each imager would create when viewing laterally.